High-precision vision-based mobile augmented reality system for context-aware architectural, engineering, construction and facility management (AEC/FM) applications

Background: Many context-aware techniques have been proposed to deliver cyber-information, such as project specifications or drawings, to on-site users by intelligently interpreting their environment. However, these techniques primarily rely on RF-based location tracking technologies (e.g., GPS or WLAN), which typically do not provide sufficient precision in congested construction sites or require additional hardware and custom mobile devices. Method: This paper presents a new vision-based mobile augmented reality system that allows field personnel to query and access 3D cyber-information on-site by using photographs taken from standard mobile devices. The system does not require any location tracking modules, external hardware attachments, and/or optical fiducial markers for localizing a user’s position. Rather, the user’s location and orientation are purely derived by comparing images from the user’s mobile device to a 3D point cloud model generated from a set of pre-collected site photographs. Results: The experimental results show that 1) the underlying 3D reconstruction module of the system generates complete 3D point cloud models of target scene, and is up to 35 times faster than other state-of-the-art Structure-from-Motion (SfM) algorithms, 2) the localization time takes at most few seconds in actual construction site. Conclusion: The localization speed and empirical accuracy of the system provides the ability to use the system on real-world construction sites. Using an actual construction case study, the perceived benefits and limitations of the proposed method for on-site context-aware applications are discussed in detail. Background Automated, on-demand, and inexpensive access to project information on-site has significant potential to improve decision-making during construction or facility management activities. This information, which is usually in the form of specifications, drawings, or schedule information, enables prompt identification, processing, and communication of discrepancies between actual and expected performance. Fast access to this information also helps project managers to proactively decide on corrective actions and minimize the cost and delays due to performance discrepancies (Golparvar-Fard et al. 2012). *Correspondence: [email protected] 1Bradley Department of Electrical and Computer Engineering, Virginia Polytechnic Institute and State University, Blacksburg, VA, USA Full list of author information is available at the end of the article Despite the importance of on-site information access, most of current approaches to jobsite progress monitoring include manual and time consuming data collection, nonsystematic analysis and visually/spatially complex reporting (Golparvar-Fard et al. 2012; Navon and Sacks 2007). As a part of data collection and analysis, field personnel have to carry large stacks of specifications and drawings on jobsites and spend significant amount of time to write down an actual progress on paper and compare it to relevant cyber-information (Khoury and Kamat 2009). Such inefficiencies in site analysis and information gathering can cause downtime or rework and ultimately lead to schedule delays or cost overruns. In addition, the quality and timing of information access and exchange can either delay or facilitate successful execution of on-site activities (Chen and Kamara 2011). © 2013 Bae et al.; licensee Springer. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Bae et al. Visualization in Engineering 2013, 1:3 Page 2 of 13 http://www.viejournal.com/content/1/1/3 To minimize these inefficiencies, we have proposed a new context-aware vision-based mobile augmented reality system, Hybrid 4-Dimensional Augmented Reality (HD4AR), which identifies location and orientation of field personnel solely based on a site photograph (Bae et al. 2012). As described by Bae et al. (2012), HD4AR allows field personnel to query and access semanticallyrich 3D cyber-information and see it precisely overlaid on top of real-world imagery. HD4AR does not need RFbased tracking technologies or inertial measurements to find a user’s position. Rather, the system takes a photograph from jobsite as input and computes the location and orientation of the user’s camera using a set of computer vision algorithms. As a result, HD4AR can be used with any camera-equipped mobile device, such as a smartphone or tablet, to provide accurate on-site localization of a field engineer and thus is practical and inexpensive to use on a construction site. As shown in Figure 1, the proposed vision-based system can successfully localize a photograph even with large changes in the viewpoint of a user. Moreover, as shown in the bottom-right screenshot, the system successfully recognizes the target building from the photograph, which includes the cellular phone displaying the building, and overlays the cyber-information precisely. This photograph has different color histogram and pixel values from the photograph taken at the actual site, which can simulate the different illumination conditions of the target scene. This paper extends our prior work on HD4AR (Bae et al. 2012) in the following ways: 1) the localization speed is further increased using direct 2D-to-3D matching, 2) different image feature description methods are implemented and tested to investigate the impact of those descriptors on performance of 3D reconstruction and localization, 3) a new homography-based 3D content-creation (annotation) method, which allows a field engineer to draw and mark any building elements within the photograph, is described. The enhanced localization speed and impact of feature descriptors will be further discussed in Section ‘Results and discussion’, while 3D annotation functionality will be examined in Section ‘High-precision augmentation with HD4AR’. The remainder of this paper is organized as follows: After brief summary of related works in Section ‘Related work’, Section ‘Method: Hybrid 4-dimensional augmented reality’ presents an overview and technical approach of the HD4AR system. The details of the 3D reconstruction process that generates a 3D point cloud model from a set of unordered photographs are discussed in Section ‘3D reconstruction with HD4AR’. Section ‘High-precision augmentation with HD4AR’ presents the localization and augmentation process using a generated 3D point cloud. The new feature of the HD4AR, e.g., 3D annotation, is also discussed in this section. Section ‘Results and discussion’ presents empirical results from experiments with HD4AR and also compares the performance to other state-of-the-art Structure-fromMotion based (SfM-based) localization solutions. Finally the perceived benefits and limitations are described in Section ‘Conclusion’. Video demos and detailed performance data of HD4AR can be found at http:// www.magnum.ece.vt.edu/index.php/research-projects/ 100-hd4ar and http://raamac.cee.illinois.edu/hd4ar. Figure 1 Screenshots of the Android HD4AR client. 3D BIM information is precisely overlaid on photos From different viewpoints. (adopted from